The phrase "to leave the table" means to stop being involved in a situation or to withdraw from a discussion or argument.